Papers with compact student models
Few-Shot Anaphora Resolution in Scientific Protocols via Mixtures of In-Context Experts (2022.findings-emnlp)
Copied to clipboard
| Challenge: | In-context learning has emerged as a promising approach to resolve anaphora, but there are challenges in applying it to scientific protocols. |
| Approach: | They propose a method which combines predictions of hundreds of in-context experts and combines them to yield a 30% increase in F1 over a competitive prompt retrieval baseline. |
| Outcome: | The proposed method yields 30% increase in F1 score over a competitive prompt retrieval baseline. |
PLaD: Preference-based Large Language Model Distillation with Pseudo-Preference Pairs (2024.findings-acl)
Copied to clipboard
Rongzhi Zhang, Jiaming Shen, Tianqi Liu, Haorui Wang, Zhen Qin, Feng Han, Jialu Liu, Simon Baumgartner, Michael Bendersky, Chao Zhang
| Challenge: | Knowledge distillation (KD) is a technique for transferring expertise from large teacher models to compact student models with reduced memory footprints and inference costs. |
| Approach: | They propose to transfer knowledge from large teacher models to compact student models by exploiting teacher-student capacity discrepancies to generate pseudo-preference pairs where teacher outputs are preferred over student outputs. |
| Outcome: | The proposed framework exploits teacher-student capacity discrepancy to generate pseudo-preference pairs where teacher outputs are preferred over student outputs. |